Progress Log

Manuel

​

This week, I first focused on hand detection using MediaPipe. MediaPipe is an open source framework from Google that to create machine learning. With it comes the ability to detect multiple hands and create a nodal frame of each hand. The following image shows the guide for each node of the hand we are able to detect and track.

​

​

​

​

​

​

​

​

​

​

​

​

We can easily install MediaPipe into our python environment using the following command in a pip enabled command prompt:

pip install mediapipe opencv-python

​

We can then go into python and import MediaPipe and use it's functions, and then create our hand and drawing variables using the following code.

​

import mediapipe as mp

mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands

​

The variable mp_hands will be used to detect the actual hand from a frame, and the mp_drawing variable will be used to draw the overlay on top of our input frame. The detection will automatically detect the multiple hands and write it into an array called multi_hand_landmarks, and then classify the landmarks as left or right handed. We can then access the classification, label and coordinates for us to manipulate, as well as the accuracy percentage, and in this case we can draw a label on the 'WRIST' landmark and overlay it on the frame.

​

def get_label(index, hand, results):
    output = None
    for idx, classification in enumerate(results.multi_handedness):
        if classification.classification[0].index == index:
            
            # Process results
            label = classification.classification[0].label
            score = classification.classification[0].score
            text = '{} {}'.format(label, round(score, 2))
            
            # Extract Coordinates
            coords = tuple(np.multiply(
                np.array((hand.landmark[mp_hands.HandLandmark.WRIST].x, hand.landmark[mp_hands.HandLandmark.WRIST].y)),
            [640,480]).astype(int))
            
            output = text, coords
            
    return output

​

With this function we can now instantiate the mp_hands detection into a while loop to detect our own hands using a webcam.

​

import cv

cap = cv2.VideoCapture(0)

with mp_hands.Hands(min_detection_confidence=0.8, min_tracking_confidence=0.5) as hands: 
    while cap.isOpened():
        ret, frame = cap.read()
        
        # BGR 2 RGB
        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        
        # Flip on horizontal
        image = cv2.flip(image, 1)
        
        # Set flag
        image.flags.writeable = False
        
        # Detections
        results = hands.process(image)
        
        # Set flag to true
        image.flags.writeable = True
        
        # RGB 2 BGR
        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
        
        # Detections
        #print(results)
        
        # Rendering results
        if results.multi_hand_landmarks:
            for num, hand in enumerate(results.multi_hand_landmarks):
                mp_drawing.draw_landmarks(image, hand, mp_hands.HAND_CONNECTIONS, 
                                        mp_drawing.DrawingSpec(color=(121, 22, 76), thickness=2, circle_radius=4),
                                        mp_drawing.DrawingSpec(color=(250, 44, 250), thickness=2, circle_radius=2),
                                         )
                
                # Render left or right detection
                if get_label(num, hand, results):
                    text, coord = get_label(num, hand, results)
                    cv2.putText(image, text, coord, cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
            
        # Save our image    
        #cv2.imwrite(os.path.join('Output Images', '{}.jpg'.format(uuid.uuid1())), image)
        cv2.imshow('Hand Tracking', image)

        if cv2.waitKey(10) & 0xFF == ord('q'):
            break

cap.release()
cv2.destroyAllWindows()

​

​

​

​

​

​

​

​

​

​

​

​

​

 

 

 

 

 

 

My next goal was to improve on our current model, and train more logos into it, using the dataset Anthony was able to gather, I was able to create a new model that can detect 14 logos. This model however while it works well with some logos, struggles with others, as well as struggling to detect with certain angles. The following images are evaluation images used to determine the model's accuracy. On the left is our model's detection and on the right is what it should be detecting as that is a manually labeled image.

​

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

​

​

​

 

​

​

​

​

​

​

​

As you can see, the model is able to detect the logos, as long as they are head on and not rotated, and the accuracy decreases the more the object is moved. Originally, I was creating the model using the SSD_Mobilenet model as a baseline, which is a unified network for object detection pre-trained model on the 2017 coco database with the ability to train for object detection. That model I was using was 320x320 which means it would convert the input image to a resolution of 320x320 and then run detection. I am now testing and using a Centernet_Resnet50 model with a resolution of 512x512, this is a slower detection model but the accuracy increases a total of 54 percent when compared to the previously used model. This might not be the final base model used, as I will continue testing different variations for speed and accuracy.

​

My next step was to combine the hand detection and logo detection. This is done by running each model individually, we first run the hand detection and then pass that overlayed image onto the logo model's detection process and output the final image. 

​

​

​

​

​

 

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​