When clicking on a Chart you show a button for Chart Tools. When clicking on a Table you show a button for Table Tools. Pivot Table Tools, Picture Tools, etc.
You can improve the experience by actually opening the appropriate tools ribbon. Chances are greatly more often than not, than when a user clicks on a chart, the next thing they are going to do is to open the Chart Ribbon. So do it for them. If they don't want the supplementary ribbon open then all the other ribbon tabs are visible to open anyway.