- Self-driven senior level systems architect, integrator, and administrator with managerial experience.
- Over 15 years of information technology experience with consistently increasing responsibility and position requirements.
- Esteemed by both peers and managers as a collaborative team player and natural leader.
- Demonstrated ability driving complex projects involving multiple departments and organizations to successful conclusion.
- Able to proactively work outside of domain experience through extrapolation, collaboration, research, and analysis.
- Effectively meet business needs by using a holistic approach: business needs assessment, project planning and scoping, asset management, customer relationship management, support follow-through, and proactive technical management.
- Exemplary problem solving ability: able to digest, fix and simplify complex systematic and architectural problems.
- Skilled at diagnosing network, service and kernel-level failures and in-depth performance issues across a wide range of operating systems.
- Well versed in systems level integration, migration, reporting, and process streamlining.
- Broadly experienced in a wide range of server, virtualization, storage, and network technologies.
- Familiar with multiple scripting and programming languages, regular expressions, and OO and MVC principles.
Work Experience and Major Accomplishments
Senior Q/A Automations Engineer
Joyent, San Francisco, CA
Dec 2018 – Sept 2018
- Implemented continuous integration and deployment for Linux kernel and platform image creation.
- Wrote initial canary services for new prototype cloud management implementation.
- Defined and implemented tests identifying Illumos kernel performance limitations and performance related bugs.
- Implemented toolchain for repeatable, comparable performance testing across multiple container and host operating systems.
Cloud Operations Engineer
Aug 2017 – Dec 2018
Joyent, San Francisco, CA
- Executed system deployment and maintanance of company's public and privately operated cloud. Participated in on-call rotation.
- Wrote heuristics to automate datacenter physical disk identification and replacement procedures.
- Authored internal specification documentation on automated deployment and testing to production.
- Identified and resolved long-standing infrastructure contingency backup performance issues. Reduced backend time for billing operations by 500%.
- Proposed new procedures and coordinated efforts to improve technical escalation to engineering.
L3 Support Engineer
Aug 2015 – Jan 2017
Nexenta Systems, Santa Clara, CA
- Chosen by NexentaStor 5 engineering team as operations’s primary technical liaison to determine engineering requirements for sustaining engineering. Identified key weaknesses in go-live strategy, specifically around command line syntax/usability and customer diagnosis engine. Worked closely with Engineering Director to ensure clear understanding, buy-in, and ability to implement the needed changes.
- Routinely selected as lead engineer or point of contact for escalated support cases with high potential for customer dissatisfaction or technical confusion. At least 95% of these cases were successfully resolved without further (management) escalation, and with either a clear technical solution applied or appropriate bugs filed against product.
- Primary operations engineer for full-stack performance analysis and complex use-case root cause analysis. Provided customers with mitigation steps which resolved pressing problems and, where appropriate, details on other environmental changes were necessary to meet customer operational expectations.
- Frequently provided impromptu consultations with L2 to proactively address issues prior to either managerial or customer escalation leading to quicker, more effective case closure rates.
- Conceptualized, gathered requirements, designed, and delivered log repository analytics toolset, providing data mining and predictive diagnosis against 32TB of raw data. Tools are used routinely through the whole of the company’s support operations, reducing time to successful resolution by 75% on at least 85% of cases raised.
- Regular collaboration with engineering and Q/A, providing in-depth discussion and diagnosis on architectural issues (specifically, with ZFS, NFS, VAAI, CIFS, filesystem normalization, and nbmand) through successful resolution. Responsible for pushing architectural changes made in multiple releases which vastly improved product stability and performance.
- Consistent top performer: out of 13 support engineers, reported 27% of all bugs in 2015 and 28% in 2016.
- Continued maintenance of support analytic scripts and log extraction and ingestion repositories.
- Routinely reported on and provided ownership of bugs for multiple engineering teams.
Senior L2 Storage Support Engineer
Sept 2013 – Aug 2015
Nexenta Systems, Santa Clara, CA
- Served as primary communication liaison between customers, customer relations, sales, and engineering.
- Lead engineer on team of 9 providing 24/7 international support for ZFS-based storage products.
- Primary technical trainer and guidance for new L1 and L2 hires.
- Primary maintainer of internal support documentation for customer-facing support activities.
- Wrote analysis tools for existing performance log gathering tools to automate time consuming support tasks.
- Heavy emphasis on and involvement in full-stack performance analysis and root cause analysis.
- Provided initial direction and requirements for next generation of product documentation.
Managed Services Power Systems Engineer
May 2012 – Sept 2013
Sirius Computer Solutions, Omaha, NE
- Regularly sought by management and outside departments for assistance with troublesome network, storage, and performance related diagnosis and resolution. Frequently brought into projects and diagnosis efforts to provide new, fresh perspective and direction.
- Planned and coordinated system-level requirements for projects for numerous development teams.
- Wrote Perl scripts to collect, audit, clean, and merge sudoers across several thousand LPARs for an S&P 400 financial services institution, allowing for tighter security compliance for account management requirements. Migration to new sudoers files was seamless.
- Served as primary escalation contact for overnight and international support incidents.
- Primary party responsible for all Linux and AIX administration needs for clients of managed services.
- Secondary responsibilities of upgrading, maintaining, troubleshooting, monitoring, and provisioning AIX LPARs.
IT Systems Manager
March 2011 – May 2012
Mathematical Sciences Research Institute, Berkeley, CA (UptimeUSA)
- Wrote project and resource plans necessary for initial customer on-boarding. Plan maintained existing service levels while competing on optimization and cost reduction.
- Supervised and provided guidance to staff of 5 in daily administrative tasks, projects, and user requests.
- Oversaw daily operations surrounding MSRI lecture video production, member onboarding, and IT system operations.
- Met regularly with MSRI staff and Uptime management to provide clarification on time lines and assure projects remained within scope through managing expectations.
- During an unplanned facility power outage, performed fuel/power calculations to Berkeley Plant staff to allow Space Science Lab and MSRI to continue operations on diesel gensets. Calculations assured optimal genset operation through the week, assuring successful NASA rocket launch at SSL.
- Performed complete equipment audit in pursuit of data sufficient to complete FY2012 equipment purchase requirements. Planned asset depreciation and project timeline for the following 3 years.
- Planned and implemented a functionally structured and maintainable architectural re-design, resulting in a virtualization consolidation of 45+ interdependent standalone systems onto 20 physical hosts.
- Reduced service cyclomatic complexity and single points of failure through an audit of systems and their functional roles.
- Through automation, playing to individual strengths, budget re-prioritization, and optimization, reduced staffing requirements from 6 full-time employees at time of on-boarding to 2 FTEs 1 year later. For instance, eliminated legacy human-intensive encoding system with scripted ffmpeg encoding process requiring minor intervention.
- Managed 3 operationally independent networks: staff, frequently cycling academic members, and public web.
- Consolidated 3 independent LDAP and 1 NIS+ authentication systems to a single central LDAP domain.
- Migrated 1,500-user mail system from Cyrus to Dovecot.
Feb 2010 – March 2011
Boomerang Management Ent., Rapid City, SD
- Primary administrator of co-located XenServer, Linux, AIX, and FreeBSD servers; client servers across 3 states. Provided technical specification and schedule for equipment replacement in the colo. Solely responsible for specification, operation, and maintenance of all server hardware/software and data backup.
- Implemented and maintained tiered backup on FreeBSD ZFS on whitebox SuperMicro/LSI chassis.
- Observed, identified, and provided written justification for AIX storage upgrade to mitigate long-standing and unacknowledged IOPS-related performance bottlenecks.
- Complete design and sole authorship of PowerShell scripts which migrated 8,000+ files based on hierarchical medical classifications from a directory tree to SharePoint. Tools were used for months for further migrations by customer without additional assistance.
- Helped administer and improve environment with 100+ concurrent Citrix ICA clients as well as 500+ concurrent SharePoint client environment (team of 3 sysadmins).
- Administered hylafax system responsible for transmittal of 3,000 faxes a day. Eliminated long-standing transmission failure causes, reducing failure rate to approximately 4% from 8-9%.
- Complete redesign and documentation of company's colocated equipment racks.
2007 – 2012
IT Independent Contractor
- Clients include Pomeroy, Sears, IBM, Best Buy, PupSafe, AtlaGroup, Gold Pages Publishing.
- Authored detailed deployment documentation and run books, including diagrams and troubleshooting guidelines, for all clients.
- Performed backup and disaster planning and implementation for multiple small and medium organizations.
- Founded AtlaGroup, a Bay Area MSP, which operated with a positive ledger through it’s first two years. Involved in operations and planning through the first year.
Site Deployment Lead
Sept 2008 – Jan 2009
Unisys in support of the 2010 Census
- Oversaw site project implementations, assuring compliance with specifications.
- Project implementation while supervising 4-5 technicians.
- Directed installation and asset delivery for physical cabling, workstations, server racks, and VoIP system.
- Configured local VoIP systems and workstations for initial use.
- Answered to project NOC at the end of each day, verifying success weekly with my immediate project manager.
Aug 2007 – Jan 2008
West River Health Services, Hettinger, ND
- Responsible for over 150 workstations and a dozen Windows servers located across 9 remote facilities in a complex hospital/clinical software environment with multiple legacy dependencies.
- Additional responsibilities included hands-on user support, project planning, asset management and acquisition, and the creation and maintenance of documentation.
Network/Computer System Admin
Dec 2004 – June 2005
Good Samaritan Society, Sioux Falls, SD
- Built, deployed, and supported Linux storage/print servers for over 250 end user facilities country-wide.
- Maintained support database and project management portal.
2000 - 2006
HES, Huron, SD
- Performed all technology related purchasing and equipment management including printers, servers, and workstations.
- Built and maintained high-end AutoCAD systems for engineers and draftsmen.
- Resolved all technical issues experienced by employees.
Intern UNIX Administrator
Sept 2000 - May 2001
Juniata College, Huntingdon, PA
BSc in Information Technology
Colorado Technical University of Sioux Falls, SD (GPA 3.45)
Study emphasis in project management, HPC clusters and security.
Areas of Technical Specialization
- Operating systems: Windows, Linux, illumos/Solaris, FreeBSD, AIX
- Automation: HashiCorp Nomad, Packer, Terraform; Ansible, Chef, Puppet, TeamCity, Jenkins
- Filesystems: NFS, ZFS, NFS, ext*, vmfs; data recovery
- Network Storage: Storage NFSv3, NFSv4, iSCSI, Fibrechannel, CIFS (Microsoft, Solaris, and Samba)
- Performance: Able to efficiently diagnose application, network, storage, OS, and platform bound issues.
- Networking: Familiar with majority of DC and rack level equipment and concepts. Deep troubleshooting experience at all layers. iptables, pf/packetfilter, packet analysis, nmap, wireshark, snort, IPSEC, OpenVPN.
- Virtualization: kvm, OpenStack, Xen, Hyper-V, VMWare vSphere and ESX, AWS
- Infrastructure: DNS, SNMP, SMTP (exim, postfix, sendmail), cyrusIMAP, Active Directory, LDAP, puppet
- Scripting/Programming: bash, Perl, PHP, Python, SQL, CSS, PowerShell, C#. Familiarity with C, Java, MVC and ETL principles.
- Documentation: writing, planning, overview, and technical qualification.
- Backup: rsync, NetBackup, MySQL (replication), ZFS