What are the best practices for optimizing Apache Spark performance?

Share your insights on memory management, partitioning, and caching to make your Spark jobs run faster and more efficiently.