Comments on 11/4 call

1. Provide your organization’s summary on the intent of the Imbalance Reserves Mosaic Parameter Requirements initiative.

PG&E understands that this is the first step in a multistep process toward the successful implementation of the imbalance reserves product and successful launch of EDAM. PG&E appreciates CAISO for considering previous stakeholder comments for additional engagement during the Imbalance Reserves Parameter tuning process and expects to continue this collaborative effort through to launch.

We agree with CAISO’s stated intent and commend CASIO as this analysis is thorough and well presented. CAISO has presented a strong enough case to move to the next step, but PG&E believes stakeholders should have the opportunity to wholistically evaluate all of the Imbalance Reserve Tuning parameters, including mosaic model parameter values, during parallel operations and the ‘fine tuning’ process.

2. Provide your organization’s comments on whether it is felt that there is sufficient existing information on mosaic quantile regression, utilized by CAISO for forming Imbalance Reserve (IR) requirement. If no, please provide where you would like to see further information and/or training.

PG&E understands that this is the first step in a multistep process toward the successful implementation of the imbalance reserve products and successful launch of EDAM.  This analysis is thorough and well presented. CAISO has presented a strong enough case to move to the next step.

3. Provide your organization’s comments on the proposed testing approach to evaluate the Imbalance Reserve mosaic quantile regression parameters.

PG&E believes CAISO has made good progress and reached reasonable conclusions based on its presentation. However, PG&E would like clarification on slide 28 of the presentation that cautions that the FRP and IR geographic diversity benefits are not directly comparable because the modeling of each product used different footprints. Why? It would seem beneficial to try to make the footprints equivalent or similar.

4. Provide your organization’s comments on CAISO's draft recommendation of retaining initially specified sample scheme 4.

PG&E appreciates CASIO’s thorough analysis and believes that it has reached a reasonable conclusion to move on to the next step with Sample Scheme 4. However, stakeholders should have the opportunity to wholistically evaluate all of the parameters during parallel operations and the ‘fine tuning’ process.

5. Provide your organization’s comments on CAISO's draft recommendation of changing sample days from 180 days to 150 days.

CAISO has presented a strong enough case to move to the next step, but we would like CAISO to clarify the historical days split window for the 150 sample days in sample scheme 4. The graphical representation on slide 11 looks like each portion of the sample is equal in size (e.g., 75 days from T-1yr and the 75 most recent days) but we would like to confirm this assumption.

6. Provide your organization’s comments on if there are additional metrics you would like the CAISO to present when analyzing the overall performance of the mosaic methodology.

None at this time.

7. Provide your organization’s comments on whether this presentation met your expectations of the overall Imbalance Reserves Mosaic Parameter Requirements initiative? If no, please elaborate.

Yes, PG&E commends CAISO’s thorough and well-presented analysis. The stakeholder call and the opportunity to comment on this step meets our expectations for this and all future steps in the tuning and implementation process.  CAISO has presented a strong enough case to move to the next step. PG&E expects to continue this collaborative effort through to the successful implementation of the imbalance reserves product and successful launch of EDAM.

8. Provide any additional comments.

1. Provide your organization’s summary on the intent of the Imbalance Reserves Mosaic Parameter Requirements initiative.

PacifiCorp appreciates the CAISO undertaking this initiative as imbalance reserves are a key component of the EDAM. As such, setting the imbalance reserve mosaic quantile regression parameters will have significant implications on how well the imbalance reserve product performs in the EDAM.

PacifiCorp supports the CAISO’s proposal to retain sample scheme four and to reduce the sample days to 150 days.

The discussion on the imbalance reserve requirement was insightful for PacifiCorp. In particular, the estimated diversity benefit percentages were helpful to see and offered good expectations for entities joining the EDAM.

2. Provide your organization’s comments on whether it is felt that there is sufficient existing information on mosaic quantile regression, utilized by CAISO for forming Imbalance Reserve (IR) requirement. If no, please provide where you would like to see further information and/or training.

There is sufficient information on mosaic quantile regression, but PacifiCorp offers a suggestion to make it easier for stakeholders to find the information. The mosaic quantile regression has been discussed in numerous initiatives and venues related to the flexible ramping product and imbalance reserve product. The technical descriptions of the mosaic quantile regression, histogram method, and requirement threshold calculations are scattered throughout various meetings and initiatives. The same could be said about the CAISO short-term forecasting (STF) team’s descriptions of their performance metrics. PacifiCorp requests that these documents and descriptions be aggregated on the initiative webpage, or some other easily accessible webpage, so that stakeholders can easily locate the information for future reference.

3. Provide your organization’s comments on the proposed testing approach to evaluate the Imbalance Reserve mosaic quantile regression parameters.

PacifiCorp supports the CAISO’s testing approach. The Company looks forward to seeing the results of testing the dynamic threshold offsets in January.

4. Provide your organization’s comments on CAISO's draft recommendation of retaining initially specified sample scheme 4.

PacifiCorp supports retaining sample scheme four as the CAISO showed the other sampling schemes did not result in any noticeable improvements.

5. Provide your organization’s comments on CAISO's draft recommendation of changing sample days from 180 days to 150 days.

PacifiCorp supports changing the sample days to 150 days as the CAISO analysis indicated it reduced the requirement while only slightly changing the coverage.

6. Provide your organization’s comments on if there are additional metrics you would like the CAISO to present when analyzing the overall performance of the mosaic methodology.

In general, PacifiCorp likes the CAISO STF team’s use of the HScore metric because it’s an effective way to compare various testing results. It would be helpful if the CAISO STF team was able to provide a technical guide to the HScore, the metrics used, which categories the metrics are mapped to, and how the metrics are weighted. The HScore presentation that was linked in the November 4 meeting is helpful, but PacifiCorp would find a written description useful for better understanding the metric. Furthermore, it is easier to point others to a technical description rather than having them re-watch a previous presentation.

PacifiCorp also requests the CAISO to present on the exceedance and movement of the imbalance reserve requirement in the January meeting, along with the coverage. The exceedance metric will help PacifiCorp better understand the trade-offs between coverage and over-procuring imbalance reserves while the movement metric will help PacifiCorp understand how the imbalance reserve product will need to be procured in practice. As discussed in other forums with respect to the movement of the flexible ramping product requirement, large movements in the requirement can be difficult for WEIM entities to manage. PacifiCorp wants to better understand if this should be a concern for the imbalance reserve product as well.

7. Provide your organization’s comments on whether this presentation met your expectations of the overall Imbalance Reserves Mosaic Parameter Requirements initiative? If no, please elaborate.

The presentation, along with the follow-up pertaining to PacifiCorp’s imbalance reserve requirement results, did meet the Company’s expectations. PacifiCorp looks forward to future meetings on this topic and working with the STF team as the Company analyzes our individual results.

8. Provide any additional comments.

1. Provide your organization’s summary on the intent of the Imbalance Reserves Mosaic Parameter Requirements initiative.

WPTF appreciates the opportunity to provide these brief comments and commends CAISO for conducting the studies and presenting results for stakeholders to compare and contrast. However, as noted below, we request that CAISO provide additional metrics to help better understand the magnitude of coverage and variability of requirement.

2. Provide your organization’s comments on whether it is felt that there is sufficient existing information on mosaic quantile regression, utilized by CAISO for forming Imbalance Reserve (IR) requirement. If no, please provide where you would like to see further information and/or training.

WPTF believes that providing high-level training on the quantile regression approach used to set imbalance reserve requirements would be valuable. As mentioned during the meeting, several observations and correlations were noted, and to assess whether these correlations are valid, stakeholders need to understand why they occur. For example, CAISO highlights a correlation between the requirement and wind. Before evaluating whether this correlation is reasonable for setting imbalance reserves, it is essential to first understand the underlying causes.

3. Provide your organization’s comments on the proposed testing approach to evaluate the Imbalance Reserve mosaic quantile regression parameters.

No comment at this time.

4. Provide your organization’s comments on CAISO's draft recommendation of retaining initially specified sample scheme 4.

Based on the results provided to date, retaining scheme 4 makes sense. However, WPTF has two additional asks before formalizing an opinion. First, WPTF would like to see the additional metrics as noted in response to #6.  Second, WPTF wonders if the CAISO has considered a sampling scheme that takes the same dates from the current and previous year. Based on the illustration of schemes on slide 11 from the presentation, it appears that sample scheme 4 takes the most recent 75 days (assuming 150 day sample set) and 75 days starting from the most recent day of the prior year going forward. For example, if evaluating for trade date Nov 1, 2024, it would include Aug 17, 2024 through Oct 31, 2024 and Nov 1, 2023 through Jan 15, 2024. Has the CAISO considered a sample scheme that included Aug 17, 2023 through Oct 31, 2023 and Aug 17, 2024 through Oct 31, 2024?

5. Provide your organization’s comments on CAISO's draft recommendation of changing sample days from 180 days to 150 days.

Based on the results provided to date, moving to 150 sample days makes sense. However, WPTF would first like to see the additional metrics as noted in response to #6 before formalizing an opinion. 

6. Provide your organization’s comments on if there are additional metrics you would like the CAISO to present when analyzing the overall performance of the mosaic methodology.

WPTF respectfully requests that CAISO provide additional metrics to better capture the magnitude of over- and under-procurement. As explained during the call, the coverage metric only quantifies how many intervals had an imbalance reserve requirement greater than the need, without providing insight into the scale of over- and under-procurement. This information is crucial for assessing cost and efficiency. For instance, the coverage metric would treat an interval with a 1,000 MW requirement and a 200 MW actual need the same as an interval with a 1,000 MW requirement and a 999 MW need. However, the first interval procured 800 MW more than necessary, leading to potentially unnecessary costs. It would also be important to understand the magnitude of under-procurement as that relates to the improved ability to reliably operate the grid in real-time.

Therefore, we respectfully request metrics that show the magnitude of over- and under-procurement on an interval-by-interval basis to help determine if the requirement needs further refinement.

Additionally, WPTF would like to see how the requirement may change throughout the day. As noted by WPTF back in March 2023, large hourly fluctuations in the requirement may occur as a result of using this approach, particularly with high levels of renewable resources. Thus, we request that CAISO provide sample days, ideally covering all four seasons, showing the hourly requirement using the mosaic quantile approach.

7. Provide your organization’s comments on whether this presentation met your expectations of the overall Imbalance Reserves Mosaic Parameter Requirements initiative? If no, please elaborate.

No comment at this time.

8. Provide any additional comments.

If the additional metrics show that the magnitude of over-procurement is quite significant and/or there are large swings in the requirement hour to hour, it may make sense to consider if the requirement needs to be further adjusted to help reduce over-procurement and/or smooth out the variability of requirement across the day.  

Additionally, WPTF asks that the CAISO make transparent the sampling scheme and days in the sample in the BPM and provide ample notification prior to any changes through market notices, similar to the CAISO's committment to provide market notification whenever the deployment parameters and/or constraints enforced in the deployment scenarios change.