Showcase: AgileX achieves two-arm collaborative tasks based on Mobile Aloha
Mobile Aloha is a whole-body remote operation data collection system developed by Zipeng Fu, Tony Z. Zhao, and Chelsea Finn from Stanford University. link.
Based on Mobile Aloha, AgileX developed Cobot Magic, which can achieve the complete code of Mobile Aloha, with higher configurations and lower costs, and is equipped with larger-load robotic arms and high-computing power industrial computers. For more details about Cobot Magic please check the AgileX website
https://preview.redd.it/9mcp67vmcy0d1.jpg?width=900&format=pjpg&auto=webp&s=f62c8fe8aad2e954537c6ce16142e10594173fbc
In previous projects, AgileX successfully implemented the entire process of data collection, data replay, model inference and reproduction for the single-arm gripping task based on Mobile Aloha. Now AgileX collects more and more complex data and achieves the entire reasoning process of the dual-arm long sequence multi-target gripping task.

Task description

The task can be described as: First, stretch out the right arm, pick up the black block from the table, and then place the black block on the horizontally placed box in the center of the table. Then place the right robotic arm back, and at the same time stretch out the left robotic arm, pick up the red block from the table, and place it on the central box on the desktop. Finally, put the left arm back in its original place.
https://i.redd.it/7x3jmj3b4y0d1.gif
Compared with the previous tasks, the difficulty of this task has been upgraded: from the original short sequence to a long sequence task. Upgrading from a single robotic arm mission to a dual robotic arm mission. The target of clamping also changes from single to multiple.

Data Collection

In this task, the Orbbec DaBai camera is used to collect at a frequency of 30HZ, and contains 50 sets of acquisition data, and each set of data is collected at a fixed step size. Camera data contains color images, depth images, and point cloud information. The data collection platform is equipped with 2 master arms and 2 followepuppet arms.
https://i.redd.it/tlzko9td4y0d1.gif

Data Replay

The data replay script loads and reads the collected joint data and reproduces it as it is.
https://i.redd.it/g9510p8g4y0d1.gif

Inference

Aloha is implemented based on the ACT (Action Chunking with Transformers) algorithm model. See the figure below for the specific model
https://preview.redd.it/506vwbys4y0d1.png?width=1131&format=png&auto=webp&s=551b148f7af0581b2528980d87abdf4458317a3b
Perform model training and inference. Pick up the black block correctly with your right arm and place it on the middle box of the table. The left arm also successfully picked up and placed the red block.
https://i.redd.it/1h6disg05y0d1.gif
The black and red blocks were manually placed back from the central box on the table. The robotic arm still recognized that the object on the box had been removed and successfully completed the task again: the right and left arms respectively picked up the black and red blocks and placed them back on the central box on the table.
https://i.redd.it/5a0fr7r25y0d1.gif

Generalization

Add interference to the original data set task actions to test the generalization ability of the model. After the left arm finished a whole task and went back to the initial place, the red black on the central box was then manually removed and placed back on the table. The right arm detected that the black block didn’t change its position, so no picking action was performed. The left arm found that the red bolck was removed, so picked it onto the central box.
https://i.redd.it/qy6fvv3s5y0d1.gif

Summary

In this case, AgileX successfully achieved the entire process of data collection, model training and inference, and generalization ability verification for a two-arm multi-target grasping task based on Mobile Aloha. AgileX will continue to collect more scenes and more complex tasks, so please stay tuned.

About AgileX

Established in 2016, AgileX Robotics is a leading manufacturer of mobile robot platforms and a provider of unmanned system solutions. The company specializes in independently developed multi-mode wheeled and tracked wire-controlled chassis technology and has obtained multiple international certifications. AgileX Robotics offers users self-developed innovative application solutions such as autonomous driving, mobile grasping, and navigation positioning, helping users in various industries achieve automation. Additionally, AgileX Robotics has introduced research and education software and hardware products related to machine learning, embodied intelligence, and visual algorithms. The company works closely with research and educational institutions to promote robotics technology teaching and innovation.

Currently taking the FCC Machine Learning Course.
I dont know how to correctly interpret the probabilities of the Model.predict function output. The CNN is meant to determine whether is an image of a cat or a dog. Some probabilities are negative and very low. I dont know how to interpret that data. Also mention the model achieves its goal with an aceptable margin. Here an example:
https://preview.redd.it/yyss5v202y0d1.png?width=759&format=png&auto=webp&s=b1141a6e7150fcfdf0b65dc2119de956a9fc8ec8
Here the code.
Link of images:

wget https://cdn.freecodecamp.org/project-data/cats-and-dogs/cats_and_dogs.zip 

Code:

# 3 train_image_generator = ImageDataGenerator(rescale=1./255) validation_image_generator = ImageDataGenerator(rescale=1./255) test_image_generator = ImageDataGenerator(rescale=1./255) train_data_gen = train_image_generator.flow_from_directory( train_dir, target_size=(IMG_HEIGHT,IMG_WIDTH), batch_size = batch_size, class_mode = 'binary') val_data_gen = validation_image_generator.flow_from_directory( directory = validation_dir, target_size=(IMG_HEIGHT,IMG_WIDTH), batch_size = batch_size, class_mode = 'binary') test_data_gen = test_image_generator.flow_from_directory( directory=test_dir, target_size=(IMG_HEIGHT,IMG_WIDTH), batch_size = batch_size, class_mode = 'binary', shuffle=False) # 4 def plotImages(images_arr, probabilities = False): fig, axes = plt.subplots(len(images_arr), 1, figsize=(5,len(images_arr) * 3)) if probabilities is False: for img, ax in zip( images_arr, axes): ax.imshow(img) ax.axis('off') else: for img, probability, ax in zip( images_arr, probabilities, axes): ax.imshow(img) ax.axis('off') if probability > 0.5: ax.set_title("%.2f" % (probability*100) + "% dog") else: ax.set_title("%.2f" % ((1-probability)*100) + "% cat") plt.show() sample_training_images, _ = next(train_data_gen) plotImages(sample_training_images[:5]) # 5 train_image_generator = train_image_generator = ImageDataGenerator( rotation_range = 360, horizontal_flip = True, vertical_flip = True, zoom_range = 0.2, shear_range = 60, rescale=1./255) # 6 train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size, directory=train_dir, target_size=(IMG_HEIGHT, IMG_WIDTH), class_mode='binary') augmented_images = [train_data_gen[0][0][0] for i in range(5)] plotImages(augmented_images) # 7 model = Sequential() model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3))) model.add(MaxPooling2D((2, 2))) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(MaxPooling2D((2, 2))) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(Flatten()) model.add(Dense(64, activation='relu')) model.add(Dense(2)) model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy']) model.summary() # 8 history = model.fit(x = train_data_gen, epochs = epochs, validation_data = val_data_gen) acc = history.history['accuracy'] print(acc) # 9 acc = history.history['accuracy'] val_acc = history.history['val_accuracy'] loss = history.history['loss'] val_loss = history.history['val_loss'] epochs_range = range(epochs) print(epochs_range) print(acc) plt.figure(figsize=(8, 8)) plt.subplot(1, 2, 1) plt.plot(epochs_range, acc, label='Training Accuracy') plt.plot(epochs_range, val_acc, label='Validation Accuracy') plt.legend(loc='lower right') plt.title('Training and Validation Accuracy') plt.subplot(1, 2, 2) plt.plot(epochs_range, loss, label='Training Loss') plt.plot(epochs_range, val_loss, label='Validation Loss') plt.legend(loc='upper right') plt.title('Training and Validation Loss') plt.show() #10 probabilities = model.predict(test_data_gen) print(probabilities) probabilities = np.argmax(probabilities, axis = 1) sample_test_images, _ = next(test_data_gen) plotImages(sample_test_images, probabilities=probabilities) # 11 answers = [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0] correct = 0 for probability, answer in zip(probabilities, answers): if round(probability) == answer: correct +=1 percentage_identified = (correct / len(answers)) * 100 passed_challenge = percentage_identified >= 63 print(f"Your model correctly identified {round(percentage_identified, 2)}% of the images of cats and dogs.") if passed_challenge: print("You passed the challenge!") else: print("You haven't passed yet. Your model should identify at least 63% of the images. Keep trying. You will get it!")