Mathematics & Engineering
Meticulously designing solutions for mathematics to meet engineering, I worked on collaborative research projects with Machine Learning Algorithm in an effort to meet engineering challenges. I believe that collaboration is the key success in research. Given that Machine Learning necessitates a statistical and probability-oriented approach to enhance performance, my engagement in related research has equipped me with the ability to apply mathematical principles effectively in my professional endeavors.
Autonomous Driving with Language Models
I conducted research at Northwestern IDEAS Lab under the guidance of Professor Zhu Qi. This invaluable experience ignited my fascination with Large Language Models (LLM) concerning human interaction, advanced reasoning abilities, and extensive insights into autonomous vehicles (AV). Specifically, the LLM facilitates behavior-level decisions, such as lane-keeping and changes, by comprehending scenes through textual descriptions. The pipeline operates by integrating a safety verifier for the proposed control input, generated with the assistance of LLM, into the AV system.
I endeavored to enhance the GPT-4 Large Language Model (LLM) by incorporating a state machine into the decision-making process. The LLM now determines state transitions using predefined rules and inferred information. As part of my dedication to this project, I have devoted my free time to acquiring knowledge on optimizing the Reflective Module. This module is responsible for monitoring the LLM and facilitating in-context learning if any violations of the state transitions occur.
3D Rendering
Research with Professor Beerel's Computing Group at USC, our objective is to devise a novel algorithm for state-of-the-art 3D rendering by leveraging 3D Gaussian point clouds. Our endeavor seeks to elevate both the quality and speed of rendering. Over the course of 5 months dedicated to this project, I delved into the realm of scientific computing. Within this initiative, the accelerated framework of GPUs, coupled with the capability to incorporate custom CUDA kernels, empowered us to deploy Stochastic Gradient Descent for optimization, achieving state-of-the-art results.
To actively contribute to the project, I embarked on a self-learning journey encompassing PyTorch, Gaussian-splatting technique, and CUDA software for GPU utilization. My responsibilities involved working on the minimization of the neural network's loss function through back-propagation and harnessing a rapid sorting algorithm executed on the GPU.
Our team trained and evaluated mainstream detection models with our synthetically-generated paired GS and RS datasets to ascertain whether there exists a significant difference in detection accuracy between these two shutter modalities, especially when capturing low-speed objects (e.g., pedestrians). I contriburted to the project by generating rolling shuttering effect on 3D scenes and identify pedestrains with Yolo object detection format.
Optical Aberrations Correction via Imaging Simulation
During the summer of 2023, I collaborated with Professor Hongwei Chen from Tsinghua University on a research project. Our focus was on leveraging a deformable neural network to eliminate the Point Spread Function (PSF) and enhance the quality of post-processed images. To generate sufficient training data for the neural network, my efforts were directed towards simulating PSF aberration images using gamma decompression and partitioned convolution. This experience led me to delve into areas such as computer vision using PyTorch and imaging classification.
Interestingly enough, Professor Hongwei Chen planned to commercialize our research findings by May. He motivated us to develop a high-speed camera incorporating the machine learning de-aberration process we have explored, aiming to produce low-storage images and videos.
Website to the company: https://www.metacam.tech/
Citation: Chen, S. , Optical Aberrations Correction in Postprocessing Using Imaging Simulation
The snippet of code on the right shows my developed algorithm to construct PSF matrix from coordinates of object point (x0 ,y0, z0), coordinates of virtual exit pupil z, coordinates of image plane z, sample range on virtual exit pupil x'_r × y'_r , sampleinterval of virtual exit pupil τ , sample range on image plane x"_r × y"_r , sample interval of sampling range at image plane τ , normal unit vector of virtual exit pupil n.
