Summary
This effort has the goals of Increasing customer stability & Performance by moving Splunk Environment into purpose-built AWS cloud for CFPB Consumer Financial Protection Bureau by IT Concepts (ITC). This effort was part of a larger SOW that covers the operations, improvement, and management of CFPB’s infrastructure environment.
Project Goals
The summarized goals of this project were as follows:
- Consolidate Multiple Splunk instances (4 separate instances) and their data flow into a Single ‘Consolidated’ Instance along with their data streams, automation, AI/ML tools, configuration management and optimize the SIEM (Security Incident and Event Management) System
- Increase Overall data retention on all data sets to 1 Year (searchable) and up to 6 years (frozen) while also increasing cost efficiency and scalability.
- Allow for more flexible scaling of the overall Splunk Infrastructure and allow for the ability to increase the functionality of the tool/platform by making access to Compute / Storage / Network resources more available and easier to access all while reducing cost.
An outline of the steps we took to complete this task:
- Have Internal Architecture design teams meet with both Splunk Sr. Architects, and AWS Sr. Architects to determine the best path forward. (EC2 instance count, storage types, IOPS calculations, etc..)
- Leverage the assistance of Vendor staff to design a custom implementation for CFPB’s specific purposes and license.
- Determine the best location within our Multi-Account Environment for this deployment that meets the current security constraints of our AWS Multi-Account.
- Deploy Splunk to Ec2 instances that leverage S3 storage (SmartStore) and confirm functionality.
- Replicate data from our production on-prem infrastructure to the new AWS Multi-Account Infrastructure.
- Deploy ML/AI tool infrastructure to AWS
- Migrate all data input points to their new AWS location.
- Decommission old on-prem infrastructure.
Due to cost, we rolled this out in a 3 phase approach in order to ‘ramp up’ to the full cost and functionality of the system.
All of these Steps can be broken down to the three categories. Those categories are
- Planning/Approvals
- Implementation
- Decommissioning
Governance
SOP’s dictate that all login to AWS routes through both Okta and Cyberark (PIV Cards). Without a PIV card, you can’t the AWS Console at all.
- CloudTrail Logging is enabled through Control Tower, and those logs are all sent to Splunk for all accounts within the Multi-Account (12 accounts)
- S3 Bucket access is not only restricted by IAM role, but also by IP address in all accounts. all access to S3 buckets MUST be approved by a standard access process (usually a PUA)
- MFA Is not currently enabled, though this is due to a conflict between Cyberark, Okta and Azure AD though this is on the roadmap and is being actively pursued.
- Users access is guided by our standard PUA process (priviledged user access), so no one gets access to any AWS or Splunk account without a PUA and 4 levels of approval including service accounts. Those 4 levels are Supervisor, Environment Owner, System Owner, Security Oversight
IT also follows AWS’s recommended security best practices:
- ACCT.01 – Set account-level contacts to valid email distribution lists
- ACCT.02 – Restrict use of the root user
- ACCT.03 – Configure console access for each user
- ACCT.04 – Assign permissions
- ACCT.06 – Enforce a password policy
- ACCT.07 – Deliver CloudTrail logs to a protected S3 bucket
- ACCT.08 – Prevent public access to private S3 buckets
- ACCT.09 – Delete unused VPCs, subnets, and security groups
- ACCT.10 – Configure AWS Budgets to monitor your spending
- ACCT.11 – Enable and respond to GuardDuty notifications
- ACCT.12 – Monitor for and resolve high-risk issues by using Trusted Advisor
ITC leverages PM Waterfall & Agile methodologies to ensure there is always a working team on all of its projects, and the PMs are required to engage with the customer lead to ensure feedback is always captured.
This takes a few forms such as:
- End of sprint reviews to ask the question “what went well, and what went wrong” and the adjust the approach.
- Weekly leadership check ins to get the higher-level perspective to ensure the effort is lining up with those higher level goals.
- Monthly Project Management (PMR) reviews that focus on compiling all gathered feedback from the various ITC teams (Sales, CTO, Support, ETC)into actionable changes for the PM leading the efforts
| Event |
Frequency |
Activities |
Value |
| Monthly Status Reports (MSRs) |
Monthly |
Consistent reporting of contract activities aligned to the program’s Statement of Work (SOW) or Performance Work Statement (PWS) is critical to monitoring and communicating project progress and results of ITC deliverables and performance aligned to:
- Quality
- Schedule
- Cost – Financials
- Management
- Business Relations
- Staffing
- Risks/Issues
- Opportunities
Teams conduct peer review of MSRs to ensure presentation consistency and content value
PMs upload monthly PMR decks into their Program SD-E |
ITC | Standardized review of MSRs ensure alignment to contract deliverables, schedule, and strategy, and that communication is fluid ensuring alignment of work activities and availability of resources; communicates issues before they become problems/risks.
Customer | MSRs provide information on the progress of a project to stakeholders. It is a synopsis of the month’s activities, and highlights changes to the project. MSRs written in alignment with the CPAR rating definition of Exceptional and Very Good communicate the value of ITC delivery consistently throughout the year and contribute to meaningful program SAs and scores. |
| Program Management Reviews (PMRs) |
Monthly |
Consistent, scheduled check-ins. Directors hold internal PMRs with PMs monthly. (EC participates in a minimum of 4 PMR reviews monthly)
- Financials
- Staffing
- Performance
- Risks/Issues
- Opportunities
- Employee Engagement
PMs upload monthly PMR decks into their Program SD-E |
ITC | Standardized review of account strategy to ensure alignment with ITC strategy, expectations, and investments; communication is fluid ensuring alignment of work activities and availability of resources; communicates issues before they become problems/risks
Customer | ITC leadership is aware of contract performance and supports the PM/team in achieving contract objectives and overcoming risks |
| In Progress Reviews (IPRs) & Impact Statements |
Quarterly |
Consistent, scheduled engagement with customers
- Review performance (good)
- Review opportunities (challenges/risks) with proposed solutions
- Review financials
- Review staffing
- Deliver Quarterly Impact Review Statement (coordinate with CxO)
|
ITC | Standardized review of account strategy to ensure alignment with ITC strategy, expectations, and investments; communication is fluid ensuring alignment of work activities and availability of resources; communicates issues before they become problems/risks
Customer | Insight into project status, performance, and resourcing; arms customers with the information they need to champion the contract and ITC monthly |
| PAR/CPAR Self-Assessment (SA) |
Annual |
DQCP schedules SA meetings 60 days before the end of each contract’s annual PoP to kick-off the SA writing. |
ITC | Provides a mechanism for PMs to discuss delivery throughout the PoP. The SA helps shape and increase the likelihood of ITC exceptional delivery documented in the government systems and process and rated in individual evaluations year over year.
Customer | Provides a venue to discuss ITC delivery and is a workload reduction tool when customers copy and paste accurate, true, aligned performance evidence that clearly articulates the value the government received as a benefit of ITC’s products and services. |
| Program Past Performance Write-up/Qual Update |
Annual |
Typically reviewed and updated in conjunction with annual SA updates occur |
ITC | Documenting annual achievements via metrics and benefit realized by the government because of ITC’s delivery increases our ability to win the recompete and additional business. Capturing the details quarterly and annually is a future workload reduction tool for ITC PMs and Proposal Writers.
Customer | Well-written Past Performance Quals helps government customers retain the ITC teams they love and want to bring back for follow-on or additional work. |
| Customer Kudos |
At time of event |
PMs and Task Leads forward customer feedback and kudos and archive in internal systems |
ITC | Collecting, storing, and retaining customer kudos helps the ITC Enterprise award strong performers and high performing teams. It also helps future proposal writers select meaningful customer quotes for future proposal. |
| Receive Performance Report |
Annually |
DQCP retrieves reports from CPARS when available or receives PARs from the PM or Director of contracts. |
ITC | Comprehensive performance reports provide ITC PMs valuable insight into how their teams are performing against requirements and focuses planning for future delivery periods. High-scoring performance reports serve as a morale booster for teams and helps shape and bolster future PP Volumes.
Customer | Required by the FAR, many customers appreciate an open dialogue and review of program SAs throughout the year because of the utility those activities provide when Assessors write annual performance reports. |
