Andrew M. Vernier

I currently work on the Azure Privileged Identity Management team where I help keep millions of users' identities secure in their day-to-day work. I am mainly focused on front-end web development which allows me to pursue my passion for creating impactful user interfaces. I am also exploring how our team can migrate from KnockoutJS to React within the constraints of internal frameworks in order to have a more performant and maintainable service. I recently redesigned our team's end to end UI test project to increase reliability and maintainability through statelessness and object-oriented paradigms such as abstraction, encapsulation, and polymorphism. Additionally, I am improving our onboarding process by developing more robust documentation, a concrete on-call training guide, and serving as an onboarding buddy and mentor for our newest engineers.

Technical skills: Typescript, KnockoutJS, React, C#, Sinon, Jasmine, Mocha, HTML, CSS, Azure Pipelines, Git

I conducted research on hybrid intelligence systems (combining human and artificial intelligence) and human-computer interaction in the Crowds and Machines (CROMA) Lab advised by Dr. Walter S. Lasecki.

My most recent project was using machine learning for conversation disentanglement which is the separation of distinct conversations that have occurred in the same thread over the same period of time. Previous work had already proven this to be possible with high accuracy however it took 255 hours of human effort to annotate the dataset and the evaluation was done on conversation within the same domain as the training data. Our goal was to learn if it was possible to obtain a similar accuracy in new domains with minimal human effort. We proposed using transfer learning to leverage the original dataset and then training the model on a small set of data in the new domain. We aimed to require users to spend on the order of one hour or less annotating data in a new domain in order to achieve high accuracy conversation disentanglement in that new domain. In order to so, we theorized that choosing the right data to annotate was critical (e.g. a single untangled conversation will likely not be as informative as a heavily entangled conversation). While it made sense that training on more entangled data was likely better, there was a number of problems with proving this that I solved.

First, I had to collect data large amounts of data about IRC channels. The primary reason for identifying channels in new domains and collecting logs from these channels. I also used this data to characterize the IRC space. For example I determined the proportion of channels that had logs (even if they weren't accessible) as we felt that if channel admins cared enough to keep around logs, disentangling those logs would be of interest to them. This was one way we motivated our research.

Second, there existed no metric for comparing the level of entanglement of threads. Thus, I conceptualized two metrics to do this - the average number of conversations at a time instant in a thread and the link intersection ratio of a thread. If we annotate a conversation, we create "links" from a message to the message that it is in response to. A link intersection occurs whenever two links span messages in the same time range (thus indicating that two conversations were occuring at overlapping times). While these metrics matched our intuition for what constitutes and more entangled conversation, I still had to prove their effectiveness. Thus, I used the strong correlation between these metrics and other metrics that we would expect to be affected by the entanglement of a thread (e.g. number of messages per minute, number of username references in messages, etc.) to motivate the usefullness of my custom metrics.

Lastly, now that I had these metrics, I was able to tackle our next problem which was proving that we could use them to determine which portions of conversations users should annotate and later train the model on to most effectively increase the accuracy of the model in the new domain. There were a number of elements to this. One was showing a strong correlation between the calculated entanglement of a thread with the original model (only transfer learning) and the trained model (transfer learning plus training in the new domain) because it would be counterproductive to annotate all the data to find the most entangled data to annotate only that data. Another important aspect was running experiments to determine the effectiveness of training on different amounts of data and various levels of entangled data. We will use these results to guide our suggestion for choosing data to annotate for new users in new domains.

Previously I also did research in the space of image annotation tools where I worked on two main projects. First, I worked on an image annotation tool that aggragated crowd worker's responses for a particle filtering task to reconstruct 3D scenes from 2D images particularly geared towards images of vehicles. This would allow us to train autonomous vehicles using already existing videos from the perspective of a car driving on the road rather than needing to go collect that data manually. This is particularly interesting because we can use videos of accidents (or "close calls") to train autonomous vehicles rather putting cars into this situation purposefully in order to collect that data. My work on this project primarily involved making improvements to the UI so that it was easier to use for our target audience - crowd workers. Initial testing with expert annotators showed that our improvements led to a 30% increase in accuracy.

The second project focused on the generalization of image annotation tools. We decided to look into what made image annotation tasks difficult and we found that there were two main types of challenges: challenges caused by the properties of the image (e.g. low color contrast) being annotated and challenges caused by the goal of the annotation task (e.g. boundary identification). This led to us developing the concepts of image property categories and task requirement categories that are found across domains of image annotation tasks. From these categories, we developed a test harness which should guide the design of image annotation tools and testbeds that wish to generalize across a range of domains. This work was published and presented at UIST 2019 and the abstract can be found here.

Technical skills: JavaScript, JQuery, HTML, CSS, Python, LaTex, PHP, Git

When I joined the Michigan Mars Rover Team (MRover), I was first beginning my career as a computer scientist. Over the first three I had grown both technically and personally through contributing to the development of the rover's codebase. A large part of my learning came through the mentorship of more senior members on the team. My last year and a half, I was one of the most seniored members on the team and I loved being able to give back to the team by passing on my knowledge to younger members. I did this through both providing one-on-one mentoring to students and by presenting technical material to the team. I was also tasked with providing feedback at design reviews and advising the new team leadership based on my past experiences on the team.

Another key responsibility of being a technical advisor was looking at improving the team from a big picture perspective. Instead of being primarily concerned with making sure our team scored as high as possible at the current year’s competition, I focused on the long term growth and success of the team. Some ways I did this was helping restructure the team where necessary as we grew, aiding the team by influencing technical and non-technical projects that allowed us to become more sophisticated and fight stagnation, and increasing member retention. One example is that I introduced new project management strategies into our workflow. We used GitHub to manage our codebase which has many built in features for managing projects such as issue tracking. Thus, it was a natural transition to start using these features with our existing projects. I first prototyped my proposed workflow with a project I was working on to get a proof of concept and then helped the software leads start using them for their projects. This improved our ability to track the progress of our work and delegate tasks to members. This also facilitated our transition to remote work during the pandemic.

I received my Master of Science in Engineering in Computer Science and Engineering from the University of Michigan in December of 2020. My coursework included Design and Analysis of Algorithms, Affective Computing, Advanced Compilers, and Computer Networks.

I have recieved the following awards for my academic performance: James B. Angell Scholar.

It's great to be a Michigan Wolverine!

Technical skills: C, Python, LaTex, Git

The Michigan Mars Rover Team (MRover) is a student run team comprised of individuals across various disciplines at the University of Michigan that designs, builds, and tests a mars rover for competition at the University Rover Challenge every summer. I joined MRover in the fall of 2017 during my sophomore year and worked with the team in various roles over my four and a half years on the team.

In my first year, I joined the Autonomous Navigation subteam where I, along with two other members, was tasked with developing the service that executed all navigation logic for the autonomous mode of the rover. The main goal was to develop code to autonomously drive our rover between marked waypoints avoiding any obstacles in our path given GPS coordinates that were up to 10 meters away from the waypoints. Under our implementation, once at the GPS coordinate we would perform a search for the waypoint marker. This was an extremely challenging project for us as neither my teammates nor I had ever worked on components in such a large project before. As we were approaching the end of my first season on the team, we had made a ton of progress however the complexity of our project had grown so much that anytime we went to add features, we would spend twice as long debugging and fixing what broke by adding the feature. After a while of this, I sat down with our chief software engineer to discuss how we could better approach the problem to avoid this issue. We decided that using a finite state machine would be a better implementation as it would allow us to model our infinitely complex task as a finite set of states. This gave us the ability to track our state transitions at every instant which allowed us to more quickly identify the set of inputs that was causing unintended behavior in our code. While this refactor was likely best for the long term, it was a large undertaking and we were quickly approaching a critical deadline. Knowing how inefficiently we were currently working, I decided to take on more responsibility and perform the refactor myself. After completing this, I gained a strong sense of ownership of the navigation service and quickly became the expert on the subject within our team. A few weeks later I earned one of the greatest feelings of accomplishment I’ve ever felt when I clicked the “on” button and saw the rover successfully turn and drive to a GPS waypoint in the back parking lot of our machine shop.

In large part because of my initiative here and my drive to keep pushing the team to build a better autonomy system, I was named the Autonomous Navigation Lead for the following season and earned myself an invitation to come to competition. Competition gave us some key insights to focus on for the next season. While autonomy mode worked extremely well in our custom built simulator, the integration with the computer vision system and odometry sensors limited our performance. Thus, I turned our focus for the following year to improving the robustness of the system in the real world despite having imperfect inputs. This led to three major improvements to the autonomous navigation system in my second year on the team. First, we improved our navigation algorithms. We gutted our search algorithm and redesigned it to work in a way that handled false-positive without discarding previous work done, and reduced false-negatives, by adding strategic pauses. These pauses made an incredible difference because, by sitting motionless for a few seconds, we gave our computer vision service a chance to view our surroundings while the rover was still (and thus the camera was stabilized). While this was a simple feature to add, it was not easy to determine it was necessary because we did not do enough testing outside of our simulator that first season. The second improvement we made was to increase the accuracy of our odometry data. To achieve this, I designed a sensor fusion algorithm to calculate location and heading estimates. The algorithm calculated multiple estimates and weigh them based on our confidence in the sensors used for each estimate. For example, instead of relying on just our IMU to calculate current heading, we obtained a second estimate through our GPS and accelerometer. We used the GPS’s track angle to determine the absolute heading while we were driving and we found the change in heading while we were turning using the accelerometer. By relying on more sensors, we became less likely to have inaccurate results in theory and this proved to be true in our testing. This also mitigated the risk of a sensor failing as we could detect a sensor failure and set the weight of the estimate(s) based on that sensor to zero. The third improvement was to decrease our reliance on a single computer vision sensor for obstacle detection. I initiated our team’s research into ultrasonic and time of flight sensors which could be used alongside our stereoscopic camera. Since these were cheaper sensors than a stereoscopic camera, it also opened the door for using multiple of them in different positions. This aided in detecting steep drop-offs that we wanted the rover to avoid. After initial research I transitioned this project to some younger members.

These efforts proved to be well worth it as our team performed very well this year at competition achieving third place in the autonomy task, our best finish in team history. More details on my role as the Autonomous Navigation Lead from a less technical perspective can be found in the Autonomous Navigation Lead section.

My third year on the team I transitioned to being the lead software technical advisor. Details about my role in this position from a less technical perspective can be found in the Software Technical Advisor section. This position also came with many technical challenges as well. First, I had to learn more about the infrastructure of our software system so that I could help maintain it. For example we use a custom build tool that manages dependencies that I add features to as the team’s needs change. Another technical project that I’ve worked on in this role is building the second version of the autonomous navigation simulator web application (picturedaboveto the right). The largest problem with our original simulator was that it had a very large learning curve and therefore to most people on the team it was a black box that people were afraid to touch. So I refactored it using a modern web framework, Vue. I also used TypeScript and ESLint in order to reduce the risk of members new to web development from making some of the common mistakes of JavaScript. This refactor also allowed me to more easily add new features to the simulator such as adding noise to our simulated sensors.

Technical skills: JavaScript, TypeScript, Vue, HTML, CSS, C++, Python, Svelte, Git