• histogram
  • chop range of values/X into partitions of fixed size - then Y/# in bin
  • choose bin width, not smooth, not unique (must choose bin anchoring)
  • problem with histogram: shape of distribution should not depend on arbitary parameter
• kernel density estimates (KDE)
  • modern alternative to histograms
  • localized, variable width, normalized (area under each curve the same), smooth, unique, must choose bandwidth
  • gaussian function to adjust bandwidth
    • large bandwidth: smooth but little detail
    • small bandwidth: noisy, but more faithful to data

• IQR …

• a point that is different from the other points
• far away from typical value
• how to measure far? - via distribution of points themselves

• a distribution that has no shape
• use logarithmic graph - makes info visible
• power law: 1/x^(1.87)
• example: earthquakes, length of rivers, stock market, …
• properites: no location nor scale; standard statistical methods do not work; occur in real world; use double-logarithmic plots to identify
• do not have good understanding into how they occur nor how to handle
• more points do not mean better accuracy; average never settles

• Alternative to density estimate but has different use
• cumulative shows fraction of points to left of current location : always increases; always smoother; useful for comparing;
• intuitive idea of shape idea of shape of function; cumulative gives quantitive idea
• properties: monotonic; unique; smoother than histogram/density; normalized to 1; use for comparison of two distribution; use to estimate tail weights

skipped

• treat sample as representation of the system
• generate bootstrap samples by sampling with replacement from the original sample
• calculate desired quantity for each bootstrap sample
• use the scatter of values to develop confidence interval for desired quantity
• summary: synthetic samples give confidence; original sample needs to be reasonably large and clean

• variant: Jackknife: "leave-one-out" resampling
• variant: parametric bootstrap: for a kernel density estimate, generate bootstrap sample from that (rather than from the original sample directly)
• Summary of resampling methods: turn point estimates into interval estimates: work if sample is decent; do not really on assumptions or analytic arguments; can be applied to non-parametric problems

…

• components: trend; seasonality; noise; other (e.g., outliers)
• tasks: description (past); forecasting (future); control (presence)
• requirements: equal-size increment; no missing points; enough data points

• task: smoothing to get rid of noise
  • solution: given a smoothing interval, replace the center point with an unweighted average of adjacent points; the larger the smoothing interval, the smoother the result
  • can lead to unrealistic artifacts (e.g., outliers cause jumps in average)
  • weighted: give greater weight to points near the center of the interval
• problems with moving averages: interior points only; no forecasting; computationally awkward (entire series must be available)

• previous smoothed value is updated based on new data
• smoothed value is mixture of new "raw" data and previous "smooth" value
• three forms
  • single : for data without trend
  • double : for data with trend
  • triple : data with trend and periodicity
• summary: simple; versatile; few assumptions; updating scheme easy to implement; use mixing parameter to balance smoothness versus faithfullness; single/double/triple depending on data set

• method
  • 1: smooth the data to establish a baseline ("location")
  • 2: find residual between baseline and data (subtract smooth value from data)
  • 3: determine distribution (via kernel density estimate) of residual to find the "width"
  • 4: use the cumulative distribution to establish "confience bands"
  • 5: outliers differ from the baseline by more than the "width"
• alternative: use hot-winters smoothing (smooth the residual)
  • advantage over outlier detection: works "up to the edge" (not just for interior points); adapts if the width of the residual changes
• special cases:
  • if baseline is known and fixed then simpler (e.g., control problems like heated room with fixed temperature setting)
  • if data is highly seasonal use "last year's value" as baseline
• summary
  • visual inspection
  • smoothing to establish baseline
  • calculate residual
  • distribution of residuals: "width"
  • cumulative distribution of residuals: "confidence bands"
  • smooth baseline to find outliers

• Ex: fluctuations around a known setpoint
• use: cusum chart: makes small change in setpoint visible

… left the workshop at this point …