Challenge 1: Privileged Data Collection
If you wonder the key challenges in a typical PAM deployment are technical, integration complexities, roll out, management? Let me share my experience on our recent and probably the (or one of the) largest enterprise Privileged Access Management (PAM) deployment in the middle east. We were awarded a very large PAM project consisting of 2500+ Servers and 1000+ Applications for a large enterprise in the region.
Having now implemented, rolled-out and successfully signed off the project, I can now tell you that the biggest challenge in a large PAM deployment is the data collection. Regardless of how well organized and managed the customer environment is, there are certain challenges in any such large deployments which I could summarize as below.
- Lack of information on how many administrators (Local/Domain) were used in each Servers/Devices
- Domain administrators without a clear privilege demarcation.
- Administrators yet to be deleted/disabled who have left the organization.
- Users who used to administrators who no longer are.
- Active directory data sanity.
- Lack of track on local administrator accounts.
- Database admin confidentiality aspect in integrating the accounts with PAM.
- Identifying Application URL's and the credentials and credential passing parameters and attributes.
- DNS resolution of URLs for web application integration.
- Lack of proper hostname/ip address inventory.
Solution
We clearly understood that our technical knowledge and project management skills alone won't help resolving these issues. So we had to take a spin on how to turn this around.
- We collaborated with the systems team to create new administrators for the servers and requested the team to cleanup the AD to remove all the administrators which were not used.
- Requested to remove all the domain administrator privileges to the previous admin users so that we could make sure that none of the admin users have the administrator privilege apart from newly created ones.
- Requested to do an audit and cleanup unwanted local administrators
- Collaborated with the network team to create new administrators for network devices and remove existing Admin privilege.
- We had organized multiple meetings with the database team so that we could collaborate with them all the more better.
- Final aspect of this deeper collaboration was to get every department's consensus to prepare their respective latest inventory list. This was the most critical piece of the puzzle and the toughest one, as the team has to get this done due diligently during their regular hectic work day. But the customer team was extremely cooperative and really understood the need of the hour.
Once the entire data set was collated, then comes the other challenges which are typical in any massive PAM deployment.
Challenge 2: Storage of monitoring/auditing records of 50 busy admins in video and
text format for 6 months
Solution
- Sizing of VMs were based on standard assumptions and made resource adjustments on production.
- Implemented as 2 node (primary and Secondary) and DR node with identical storage capacity.
- Active-Active HA Cluster synchronization was implemented in such a way that system configuration will be automatically synced and the session recording and session audit will be stored individually.
- System configuration has done from primary node and and will be replicated to other nodes.
- All the nodes are load balanced for user access.
- Audited sessions of all nodes can be access from single location.
Challenge 3: Managing around 1000+ apps sessions individually on to jump server
Solution
- Application access controlled by the jump server’s concept with isolated application client execution.
- Configured 3 jump servers with Microsoft RDS clustering and load balancing.
- Apps are configured identically on 3 Jump servers.
- Users can access application automatically without providing the target credentials and session are recorded too as per the policy set.
Challenge 4: Managing 50+ admins and few many external user sessions
Solution
- Individual user groups are created
- Resource groups are allocated based on individual target accounts
- Authorization are mapped using user group and resource group
Challenge 5: Password Management
Solution
- Protected sensitive credentials in a certified vault
- Automated management and cycling of passwords
- Full control and tracking of credential visibility
- Eliminate direct access to all systems
- Password access through an approval workflow
- Approval workflow has been defined to be authorized to access the target password
Challenge 6: Break the glass scenario
Solution
- Implemented a "breaking glass" mechanism which allows a super user (typically the highest authority in the organization or in the IT) to get the passwords of the target account groups gathered in the PAM system. This may be useful in the event of a disaster scenario and is needed from enterprise DR framework.
- These passwords encrypted with a strong encryption algorithm and using a public/private key pair.
Challenge 7: User roll out
Solution
This is where client's cooperation can't be emphasized enough. We were lucky that, not just the the project management team, but the entire IT team at the client's site was very knowledgeable about the implication of PAM project and knew the seriousness very well. So had an amazing cooperation from them to successfully roll out the users and eventually sign off the project.
Conclusion Note
Technical expertise, subject matter expertise, project management skills, product knowledge, integration skills - all these are fundamental and inevitable for projects of this size. But none of them would have worked if we weren't patient enough, empathizing with the customers hectic regular work schedule, going extra miles above and beyond the defined scope of work and keeping our business mantra and philosophy that "Customer support is an attitude and customer support is everything in the industry we are in".
--
Contributed By,
Bibu Jacob
Technical Manager
CodeGreen Systems
