At the Chicago-Wisconsin RUG Fall Conference in October 2012, I gave a presentation on common Teamcenter performance issues and solutions along with listing, in one place, many other past Teamcenter performance presentations and white papers. It was very well received by a standing room crowd, so I thought I'd share it with the rest of the PLM World community. Here is a link to the presentation (you must be a TcUA SIG member to view it): http://www.plmworld.org/p/do/sd/sid=3758&type=0
For those unwilling to join the TcUA SIG to view it, I wrote an abbreviated version here:
Why Performance Matters
Productivity – doing more work with less
- Improve end user productivity
- Improves administrator productivity
- fewer help desk calls responding to or fixing preventable problems
- A well tuned Teamcenter environment also improves system stability and uptime.
- Less opportunity for data corruption
- Fewer operational errors/problems
- Cost Savings
- Less waiting means less time wasted.
Quick Case Study: Company ‘S’
Performance was so bad that something which used to take 10 minutes was taking 2 hours to do.Did pretty much everything wrong, performance-wise, at first but ultimately fixed every major problem. What took 2 hours now only takes 2 minutes.
See my March 2012 article on PLM World’s website for details:
Teamcenter Performance – Hard-Earned Lessons: http://www.plmworld.org/p/bl/ar/blogaid=152
Common Performance Bottleneck Causes
Using OOTB settings — OOTB settings are intended for development environments; not production environments.
Overloaded servers — Putting too much on a single server.
Operating system settings — Network settings most common (e.g. TCP parameters).
Lack of performance monitoring & tuning — Performance doesn’t improve all by itself. It requires human intervention to determine root cause and address it.
Databases – The Most Likely Culprit
- Database performance is highly sensitive to hardware, software, and DB configuration.
- DBAs at companies generally don’t do performance monitoring and tuning of your Teamcenter DB.
- It’s typically the last place that Teamcenter administrators look for performance issues rather than the first.
Mitigating Database Server Bottlenecks
- Must use dedicated DB server
- Do not use your DB server for anything other than your Teamcenter production database. This includes not serving additional databases from the same server.
- Must have fast dedicated storage for DB files
- Avoid using a NAS device; especially a shared NAS. Use DAS (Direct Attached Storage) instead.
- Use multiple fast disk spindles partitioned with proper RAID levels according to the DB vendor & SPLM recommendations.
- Split DB data, temp, and log files across those RAID partitions according to DB vendor & SPLM recommendations.
- Use disk controllers with a battery/flash backed cache.
- Use multiple disk controllers if possible.
- Cram the RAM
- Ideally should be greater than the database size or in-memory footprint. Otherwise disk swapping/paging will occur.
- Use 64-bit OS & DB software
- 32-bit software has severe limitations regardless of whether you use “/3G”, AWE, or PAE settings to access memory beyond 3-4G. It’s still a bottleneck.
- Use a good quality network adapter(s)
- Often overlooked as potential source of bottleneck (e.g. packets/sec limitations). Consider multiple NICs “trunked” for better throughput.
- DB maintenance tasks
- Update statistics and rebuild indexes regularly.
Common Teamcenter Server Bottleneck Causes
- Overloaded Tc Servers
- Poor Web Tier Configuration
- Poor FMS Configuration
- Debugging Turned ON
- Rich Client using OOTB settings
‘Unload’ Overloaded Teamcenter Servers
- Put Dispatcher modules on separate computers away from other Tc Servers.
- Separate the Web & Enterprise tiers from the Tc Corporate and Volume servers.
- Use load balancing for Middle Tier & FMS:
- Use multiple Web/Enterprise Tier servers to open up potential hardware resource bottlenecks (e.g. CPU, RAM, network adapter I/O, disk I/O, MB bus, etc.).
- Set up multiple FSC cache servers to take load off busy volume servers/storage.
Web Tier Configuration
- Do not use port 80 or 8080
- HTTP traffic on those ports is considered web browsing traffic and is therefore given lowest priority on any network. Can also cause randomly dropped connections.
- Enable http compression if not using WAN acceleration
- Must be done on both web server and on clients to take effect.
- Change/tune OOTB settings for Web Server/App
- E.g. timeout values, max # of threads, Java memory, etc.
- Read tuning guides specific to your chosen web server/app
- Scale it up or sideways
- Add another web tier server, or increase # of work processes (e.g. Web Garden mode for IIS).
Common FMS Bottleneck Causes
- Data improperly routed
- E.g. Forcing data to go through a remote FSC server over a WAN and back again over the same WAN instead of simply pulling it directly from a nearby Volume/cache server.
- Using OOTB settings
- For development purposes only, remember?
- Missing client IP address subnets
- No load balancing
- Not using remote cache servers for WAN users
- Not using remote volume servers for WAN users
- Ensure routing is correct
- Between multiple FSC groups via entry/exit/link parameters
- Between clients and their assigned FSC servers by using complete and accurate ‘clientmap’ parameters for ALL clients.
- Use correct transport algorithms for LAN and WAN users.
- Tune FSC/FCC cache settings
- Simply picking a total read/write cache size is not enough.
- Read “Sizing the FMS fast cache” in the Teamcenter help documentation. There is also a FMS cache sizing tool available from the GTAC web site. Link to FMS Calculator
- Ensure correct client maps
- Determine ALL potential client IP address ranges and where they are located. Don’t forget to account for VPN IP addresses.
- Place client maps in the closest FSC group to their location.
- Use multiple FSCs for load balancing
- Use “front-end” FSC cache servers to reduce load on Volume servers. (Requires disabling of FSC direct routing)
- Use redundant FSC cache servers to load balance each other. Also provides fail-over.
- Place FSC cache servers close to users
- Greatly improves read performance (load time) over a WAN for groups consuming shared data.
- Place Volume servers close to users
- Use Store & Forward or at least place remote volumes near remote users. (Backups of remote volumes are critical)
- Greatly improve write performance (save time) over a WAN especially for CAD data.
- Prepopulate FSC caches
- Run a nightly script to prepopulate shared data across FSC caches.
Misc Teamcenter Changes
- Disable logging and other debugging tools. Turn them on only when needed. Examples,
- Rich Client ‘Teamcenter.ini’ configuration:
- Modify Teamcenter.ini file to increase Java Memory and other settings. Will improve performance/stability when perform large operations (CAD, PSE expansion, large Workflows).
- Enable FCC File Warming
- Improve startup time by having FCC and Tc files pre-loaded at OS login.
- Use a hardware-based WAN accelerator if using Teamcenter between multiple facilities over a WAN.
- If none then make best use of software-based Web server and FMS WAN acceleration settings. (i.e. http compression & FMS compression)
- Optimize OS network settings
- OOTB settings are insufficient
- Applies to both servers and clients
- Examples, increasing # of tcp ports, disabling NAGLE algorithm, increasing buffer sizes, trunking multiple NICs, enabling Compound TCP, Large Windows Scaling
Recommended Server Changes (Windows)
Recommended Client Changes (Windows)
Performance Monitoring Tools
- Some useful performance monitoring tools:
- Performance Monitor (comes with Windows)
- Used to monitor OS, Teamcenter, and SQL use of resources
- Wireshark (freeware network packet sniffer)
- Used to help identify network TCP performance between computers.
- Available at http://www.wireshark.org/
- Sysinternals Suite (free from Microsoft web site)
- Available on GTAC Web Site @ http://support.industrysoftware.automation.siemens.com/docs/teamcenter/
- Teamcenter Deployment Guide
- Teamcenter Network Performance Tuning
- JBoss 4.2.2 Installation & Tuning Guide
- Optimizing Teamcenter Client Performance
- Teamcenter System Performance Analysis
- Oracle documentation & web sites
- MS SQL Server documentation in addition to:
- Best Practices for Running Siemens Teamcenter on SQL Server
- Siemens Blog on Technet
- Past PLM World Conference Presentations available at www.plmworld.org,
- Teamcenter 4-Tier Deployment – Best Practices
- Teamcenter – Database Performance
- JBoss Optimization for Teamcenter
- Optimizing Teamcenter Client Performance
- Teamcenter Database Server Configuration & Tuning
Larry Carpenter P.E., PLM World Teamcenter SIG Chair, email@example.com
Teamcenter UA SIG: http://www.plmworld.org/TC_UA
Alternate Contact Info: firstname.lastname@example.org, email@example.com