Teamcenter is Slow! Teamcenter Performance Bottlenecks & Mitigation

At the Chicago-Wisconsin RUG Fall Conference in October 2012, I gave a presentation on common Teamcenter performance issues and solutions along with listing, in one place, many other past Teamcenter performance presentations and white papers. It was very well received by a standing room crowd, so I thought I'd share it with the rest of the PLM World community. Here is a link to the presentation (you must be a TcUA SIG member to view it):  http://www.plmworld.org/p/do/sd/sid=3758&type=0

For those unwilling to join the TcUA SIG to view it, I wrote an abbreviated version here:

Why Performance Matters

  • Productivity – doing more work with less
    • Improve end user productivity
    • Improves administrator productivity
      • fewer help desk calls responding to or fixing preventable problems
  • Reliability
    • A well tuned Teamcenter environment also improves system stability and uptime.
    • Less opportunity for data corruption
    • Fewer operational errors/problems
  • Cost Savings
    • Less waiting means less time wasted.

Quick Case Study: Company ‘S’

Performance was so bad that something which used to take 10 minutes was taking 2 hours to do.Did pretty much everything wrong, performance-wise, at first but ultimately fixed every major problem. What took 2 hours now only takes 2 minutes.

See my March 2012 article on PLM World’s website for details:

Teamcenter Performance – Hard-Earned Lessons:  http://www.plmworld.org/p/bl/ar/blogaid=152

Common Performance Bottleneck Causes

Using OOTB settings — OOTB settings are intended for development environments; not production environments.

Overloaded servers — Putting too much on a single server.

Operating system settings — Network settings most common (e.g. TCP parameters).

Lack of performance monitoring & tuning — Performance doesn’t improve all by itself. It requires human intervention to determine root cause and address it.

Databases – The Most Likely Culprit

  • Database performance is highly sensitive to hardware, software, and DB configuration.
  • DBAs at companies generally don’t do performance monitoring and tuning of your Teamcenter DB.
  • It’s typically the last place that Teamcenter administrators look for performance issues rather than the first.

Mitigating Database Server Bottlenecks

  • Must use dedicated DB server
    • Do not use your DB server for anything other than your Teamcenter production database. This includes not serving additional databases from the same server.
  • Must have fast dedicated storage for DB files
    • Avoid using a NAS device; especially a shared NAS. Use DAS (Direct Attached Storage) instead.
    • Use multiple fast disk spindles partitioned with proper RAID levels according to the DB vendor & SPLM recommendations.
    • Split DB data, temp, and log files across those RAID partitions according to DB vendor & SPLM recommendations.
    • Use disk controllers with a battery/flash backed cache.
    • Use multiple disk controllers if possible.
  • Cram the RAM
    • Ideally should be greater than the database size or in-memory footprint. Otherwise disk swapping/paging will occur.
  • Use 64-bit OS & DB software
    • 32-bit software has severe limitations regardless of whether you use “/3G”, AWE, or PAE settings to access memory beyond 3-4G. It’s still a bottleneck.
  • Use a good quality network adapter(s)
    • Often overlooked as potential source of bottleneck (e.g. packets/sec limitations). Consider multiple NICs “trunked” for better throughput.
  • DB maintenance tasks
    • Update statistics and rebuild indexes regularly.

Common Teamcenter Server Bottleneck Causes

  • Overloaded Tc Servers
  • Poor Web Tier Configuration
  • Poor FMS Configuration
  • Debugging Turned ON
  • Rich Client using OOTB settings
  • 

‘Unload’ Overloaded Teamcenter Servers

  • Put Dispatcher modules on separate computers away from other Tc Servers.
  • Separate the Web & Enterprise tiers from the Tc Corporate and Volume servers.
  • Use load balancing for Middle Tier & FMS:
    • Use multiple Web/Enterprise Tier servers to open up potential hardware resource bottlenecks (e.g. CPU, RAM, network adapter I/O, disk I/O, MB bus, etc.).
    • Set up multiple FSC cache servers to take load off busy volume servers/storage.

Web Tier Configuration

  • Do not use port 80 or 8080
    • HTTP traffic on those ports is considered web browsing traffic and is therefore given lowest priority on any network. Can also cause randomly dropped connections.
  • Enable http compression if not using WAN acceleration
    • Must be done on both web server and on clients to take effect.
  • Change/tune OOTB settings for Web Server/App
    • E.g. timeout values, max # of threads, Java memory, etc.
    • Read tuning guides specific to your chosen web server/app
  • Scale it up or sideways
    • Add another web tier server, or increase # of work processes (e.g. Web Garden mode for IIS).

Common FMS Bottleneck Causes

  • Data improperly routed
    • E.g. Forcing data to go through a remote FSC server over a WAN and back again over the same WAN instead of simply pulling it directly from a nearby Volume/cache server.
  • Using OOTB settings
    • For development purposes only, remember?
  • Missing client IP address subnets
  • No load balancing
  • Not using remote cache servers for WAN users
  • Not using remote volume servers for WAN users

FMS Configuration

  • Ensure routing is correct
    • Between multiple FSC groups via entry/exit/link parameters
    • Between clients and their assigned FSC servers by using complete and accurate ‘clientmap’ parameters for ALL clients.
    • Use correct transport algorithms for LAN and WAN users.
  • Tune FSC/FCC cache settings
    • Simply picking a total read/write cache size is not enough.
    • Read “Sizing the FMS fast cache” in the Teamcenter help documentation. There is also a FMS cache sizing tool available from the GTAC web site. Link to FMS Calculator
  • Ensure correct client maps
    • Determine ALL potential client IP address ranges and where they are located. Don’t forget to account for VPN IP addresses.
    • Place client maps in the closest FSC group to their location.
  • Use multiple FSCs for load balancing
    • Use “front-end” FSC cache servers to reduce load on Volume servers. (Requires disabling of FSC direct routing)
    • Use redundant FSC cache servers to load balance each other. Also provides fail-over.
  • Place FSC cache servers close to users
    • Greatly improves read performance (load time) over a WAN for groups consuming shared data.
  • Place Volume servers close to users
    • Use Store & Forward or at least place remote volumes near remote users. (Backups of remote volumes are critical)
    • Greatly improve write performance (save time) over a WAN especially for CAD data.
  • Prepopulate FSC caches
    • Run a nightly script to prepopulate shared data across FSC caches.

Misc Teamcenter Changes

  • Disable logging and other debugging tools.  Turn them on only when needed. Examples,
    • TC_SECURITY_LOGGING=OFF
    • TC_APPLICATION_LOGGING=OFF
    • TC_SLOW_SQL=-1
  • Rich Client ‘Teamcenter.ini’ configuration:
    • Modify Teamcenter.ini file to increase Java Memory and other settings. Will improve performance/stability when perform large operations (CAD, PSE expansion, large Workflows).
  • Enable FCC File Warming
    • Improve startup time by having FCC and Tc files pre-loaded at OS login.

Network Performance

  • Use a hardware-based WAN accelerator if using Teamcenter between multiple facilities over a WAN.
    • If none then make best use of software-based Web server and FMS WAN acceleration settings. (i.e. http compression & FMS compression)
  • Optimize OS network settings
    • OOTB settings are insufficient
    • Applies to both servers and clients
    • Examples, increasing # of tcp ports, disabling NAGLE algorithm, increasing buffer sizes, trunking multiple NICs, enabling Compound TCP, Large Windows Scaling

Recommended Server Changes (Windows)

See Presentation.

Recommended Client Changes (Windows)

See Presentation.

Performance Monitoring Tools

  • Some useful performance monitoring tools:
  • Performance Monitor (comes with Windows)
  • Used to monitor OS, Teamcenter, and SQL use of resources
  • Wireshark (freeware network packet sniffer)
  • Used to help identify network TCP performance between computers.
  • Available at http://www.wireshark.org/
  • Sysinternals Suite (free from Microsoft web site)
  • http://technet.microsoft.com/en-us/sysinternals/bb842062

Reference Materials

Contact Information

Larry Carpenter P.E.,  PLM World Teamcenter SIG Chair,  tcua@plmworld.org

Teamcenter UA SIG:  http://www.plmworld.org/TC_UA

LinkedIn:  http://www.linkedin.com/pub/larry-carpenter-pe/44/5b8/aaa

Alternate Contact Info: ideas2nx@plmworld.org, larry.carpenter@siemens.com

21 Comments
7 Likes

Can't access the file

October 19, 2012 12:37 PM by Patrick Kennedy

Larry, when I click the link I get a response of "You have requested information you are not authorized to view."  Any suggestions?

Access to the Presentation

November 4, 2012 12:54 PM by Larry Carpenter

You have to be a Teamcenter SIG member to have access to SIG Library Files.  You can either join the SIG to gain access, or I can email it to you.

Same error - not authozed to view

December 5, 2012 06:49 AM by Jeetendra Shakya

Hi Larry,


   It looks very informative, can you please send me in my email jeetendra.shakya@ntc-europe.co.uk.


Cheers,


Jeetendra.

Please sent presentation

August 28, 2014 02:07 AM by Rüdiger Maus

Hello Larry,
I'd also like to have the presentation sent by mail: Ruediger.Maus@Terex.com
Thanks in advance!


Rüdiger Maus, Terex Cranes Germany

PLM World access error

October 21, 2014 12:34 PM by Padmakar Neelapareddy

Dear Larry,

Excellent information... can please send me a copy of presentation to my email id below

N.Padmakar@tcs.com

 

HW specifiaction for TEAMCENTER

February 27, 2015 11:28 PM by ANNA

Hi;

May I ask what is the specification need for hardware to install a teamcenter?

 

thanks

Can't access the presentation

April 1, 2015 09:25 AM by Jasson

Hi Larry,
Good day!

Can I request for your presentation? I think it would really help me a lot.Thank you!

By the way, here's my email: jassondemapan@rocketmail.com

 

Thankyou Very Much!

JASSON

Presentation

April 6, 2015 09:28 AM by Rick Espada

I would like to have the presentation can you email it to me respada@odu.edu?

Thanks

Plz share the presentations

April 21, 2015 11:28 AM by Syed

Hi Larry,

Thanks for your efforts its looks very informative and helpfull. Can you please share the presentations on qadar4u4s@gmail.com

 

Thanks a lot.

Syed

Access to File - Hi Larry could you please email the ppt to roshan80@gmail.com as info is very good

June 3, 2015 02:33 AM by roshan

Access to the Presentation

June 10, 2015 03:42 AM by suguna subramanian

Hi Larry,

Valuable and very much informative. May I ask you to share the presentation with me (suguna22k@gmail.com) 

Many Thanks.

Suguna Subramanian

 

Please share the presentation

June 14, 2015 04:23 AM by KrishnaKumar Arunachalam

Dear Larry,

Article has excellent information. I request you to share the presentation to my below mail address.

Email Id : krishnakumar_a04@infosys.com

Access to the Presendation

December 15, 2015 01:35 PM by Duffy Toler

Hi Larry, 

Great info, can you send me the presentations as well at duffy.toler@bshg.com.  Thanks!

Content is excellent.Require the Server & Client presentations

March 28, 2016 11:10 PM by Sathesh Rajam

Hi Larry,

   It looks very informative, can you please send presentations to my email sathesh.rajam@lnttechservices.com

Thanks in advance.

Regards,

Sathesh Rajam

Presentation

June 15, 2016 11:41 AM by Jesper Frische

Hi Larry,

Very interesting.

Can I ask you please to send the presentation to:  jfrische@gmail.com

Thanks,

Jesper

Access to presentation

July 29, 2016 04:55 AM by Marcin Ladecki

Dear Larry,

could you send me the document as it seems to be very useful. Thank you in advance.

Regards,
Marcin (plm@freebox.cc)

Hi Larry ,

October 21, 2016 03:47 AM by revanth katta

Thanks for the info.

Can you please send me the presentation.

revanth.katta@gmail.com

Thanks again !

Teamcenter Slow Performance presentation

January 27, 2017 09:35 AM by Nitesh

Hi Larry

Very interested in reading your presentation, please could you email me a copy nitesh.laxmidas@bbusl.com

Many thanks, N

Presentation

March 10, 2017 04:51 PM by Ram Meda

Hi Larry,

Thanks for the nice article, please send me the presentation to rammeda@gmail.com.

Thank you very much.

Ram

PLM World Access Error

May 9, 2017 09:29 AM by idrissa sakande

Can I request for your presentation? I think it would really help me a lot.

My email: idrissa.sakande@siemens.com

Thank you!

 

Become SIG Member / Performance Presentation

September 21, 2017 02:30 AM by Michael Franz

Hi Larry,

this reads like some very interesting thoughts. How can I become a TC UA SIG Member as a German resident and access the presentation?

Thanks

Michael

Recent Stories
Voting begins for open positions on the PLM World Board of Directors

PLM World 2014 Call For Abstracts Deadline January 17th

Making Teamcenter 10.1 Licensing Work with Older Teamcenter Versions