My Career at Lawrence Livermore National Lab
I began my career at LLNL shortly after I graduated in two years from UC Davis with a B.S. Computer Science in late 2010. After almost a decade of service in my 20's, I made the difficult decision to depart in late 2019 to focus on all of my own ventures.
When I joined the ROSE team in 2010, there were just 4 other staff members. Through the years, we grew in number while also hosting many post-docs, interns, and visiting scientists from around the nation and the world.
I had never heard of the Lab prior to interviewing with them, and that's one of the main reasons I decided to join in the end. While all of my college peers were going to the typical tech companies like Google and Facebook, I decided I would use my 20's as an early opportunity to try something different and on the cutting-edge of national importance.
The Lab is a huge square campus, and placed right in the middle of it all is a beautiful lake with peaceful walking trails and a variety of exercise stations. It's also next to the Central Cafeteria and Starbucks cafe.
My first office (2010-2015) had a nice big window and was "lakeside real estate," as I liked to call it.
My second office (2015-2019) was right next to the National Ignition Facility (NIF), a massive building housing the largest and most energetic laser facility in the world. NIF is the size of a sports stadium—three football fields could fit inside. It has the ultimate goal of nuclear fusion a.k.a creating a sun here on Earth.
Wikipedia provides an excellent description of the Lab: "LLNL is self-described as a 'premiere research institution for science and technology applied to national security.' Its principal responsibility is ensuring the safety, security and reliability of the nation's nuclear weapons through the application of advanced science, engineering, and technology. The laboratory also applies its special expertise and multidisciplinary capabilities towards preventing the proliferation and use of weapons of mass destruction, bolstering homeland security, and solving other nationally important problems, including energy and environmental needs, scientific research and outreach, and economic competitiveness.
The laboratory is located on a 1 sq. mi.(2.6 km2) site at the eastern edge of Livermore. It also operates a 7,000 acres (28 km2) remote experimental test site known as Site 300, situated about 15 miles (24 km) southeast of the main lab site. LLNL has an annual budget of about $1.5 billion and a staff of roughly 8,000 employees."
ROSE: The Software
ROSE, the software project I was assigned to, at its core is a source-to-source compiler infrastructure used to build tools for analyzing, optimizing, and transforming software in languages like C/C++, Fortran, Java, even binary executables, and is extensible to support virtually any "frontend" as they are called in compiler lingo.
Some examples of what's possible with a source-to-source compiler infrastructure include detecting security vulnerabilities and bugs in software, reformatting and refactoring code, automatically rewriting code to run on supercomputers with massive amounts of parallelism, and translating code from one language to a different language.
Here's some slides I prepared to present an overview of ROSE:
I worked across the source code base from core features to bug fixes while also implementing a static analysis infrastructure tool called Compass2 that allows other developers to implement their own static analysis plugins. All of these projects utilized a wide range of languages like C/C++, Java, Python, Ruby, Perl, and shell scripting (e.g. Bash) on platforms such as Mac OS, Windows, Redhat Enterprise Linux (RHEL 5, 6, 7), CentOS, Ubuntu, Fedora, and many others.
ROSE: The Project
ROSE is a huge code base comprising millions of lines of code, including a comprehensive regression test suite, documentation, tutorials, tools, and projects. We strived to support many combinations of user and developer environments and toolchains.
All of this requires a lot of engineering for new features and quality assurance, as well as experienced project management for agile development. So as one of the only two software engineers on the team and the only certified Stanford project manager and Agile Scrum Master, my work spread across the entire spectrum of tasks.
On the software engineering front, I started by refactoring and optimizing the build systems for user and developer productivity and testing efficiency using both Autotools (Automake, Autoconf, Libtool, etc.) and Kitware's CMake.
I also architected our entire release system and accompanying infrastructure, including implementing and administering our continuous integration and delivery systems (namely, Hudson CI, later renamed to Jenkins CI) which overtime grew to dozens of jobs, many with dozens of configurations in matrix tests, utilizing over 50 in-house high-performance servers. I procured and administered all of the hardware and also eventually added the Amazon Web Services (AWS) cloud for offsite scaling and Docker for containerization.
Additionally, I implemented a variety of devops dashboards and automation tools across the Network File System (NFS) to provide a scalable, automated, and uniform environment and toolchain to increase the parity across development, testing, deployment, and end usage.
ROSE supports a constantly evolving configuration of compilers (GNU, Intel, Clang/LLVM, etc.), Boost C++ Libraries, Java, and many other dependencies, on a variety of platforms from Mac to Linux to Windows. Eventually, when the technology matured, I started to containerize ROSE using Docker.
I also implemented and managed a suite of third-party testing infrastructure, including Klocwork and Coverity for static and dynamic analysis of the ROSE source code to provide triaging and fixing of ROSE software defects.
To track and manage project activities, I used the open-sourced Redmine in the early days, but quickly moved to the Atlassian suite of tools, namely Jira, Confluence, and Portfolio. I also utilized Github and Bitbucket for issue tracking as a bridge to customers. I customized and managed all of these collaboration systems, constantly evolving them with custom workflows to suit the needs of our agile team.
In conjunction with our business manager, I aligned project Epics with overall project milestones and customer deliverables while leading weekly team meetings and managing team sprints and product backlogs. I was also constantly designing and refining our processes to streamline them for increased team productivity and satisfaction.
Through the years, I also helped ROSE transition from Subversion (SVN) to Git, Gitolite, Github, and even an internal Github Enterprise server. I managed many systems across the enterprise networks and the supercomputer networks in Livermore Computing.
To aid development, I implemented code review automation tools, policies, and practices using Google's Gerrit and in later years, Github.
Last but not least, I managed our internship program through the years and that was always a highlight for me. It was fun to meet and mentor students from all around the world. At one stage we welcomed 20 to 30 students during the summer times. Some students would even stay for 6 months to a year. It was always a breath of fresh air to work with young and talented computer scientists (mostly graduate students) on their cutting-edge research projects utilizing ROSE.
Worked hard, played harder
The Lab has a bunch of awesome "Networking Groups" where you can connect with others at the Lab, both staff and the large amount of frequent visitors and interns.
My group of choice was the Livermore Cycletrons—probably one of the nicest bicycling clubs around. Still to this day, almost a decade later, I choose to go on this group ride because there's a lot of strong riders and the courses we do Monday through Friday around Livermore are just a blast!
On Monday's, we do a Patterson Pass climbing loop while Tuesday's and Friday's are a flat "race" around the Carneal Loop. Wednesday's are usually a restful Tesla Road climb or the scenic "Tuscany" loop while Thursday's are a hard and hilly Wente Bump.
The Cylcetrons alone are probably enough to keep people working at the Lab... But the cutting-edge work and incredibly smart work force is pretty awesome too!
Leaving the Lab
I never expected to stay at the Lab as long as I did, but upon joining, I quickly saw why people tend to spend their entire careers there: Great people and unmatched culture, rewarding, cutting-edge work, and surprisingly competitive benefits and compensation!
I made the very difficult decision to leave the Lab in late 2019 as a Top Secret (Q) Cleared indefinite employee, and I'll always remember all of the incredible years I invested there and the countless memories I've shared with my team. Who knows, maybe someday I'll go back!